1. Field of the Invention
The present invention relates to a multi-eye image pickup apparatus and, more particularly, to a multi-eye phototaking method and apparatus which provide a panoramic image having an arbitrary aspect ratio by combining a plurality of images.
2. Related Background Art
As a conventional aspect ratio changing scheme of changing the aspect ratio of a television screen, an aspect ratio changing scheme of trimming the upper and lower portions or left and right portions of a screen when image data is output is known. For example, such an aspect ratio changing scheme is used to change the aspect ratio (4:3) of an NTSC television screen to the aspect ratio (16:9) of an HD or ED2 television screen.
These aspect ratio changing schemes use part of a phototaken image. For this reason, especially when an object to be photographed is phototaken by an NTSC camera using an image sensor with an aspect ratio of 16:9, and the resultant image is output to an NTSC monitor with an aspect ratio of 4:3, the horizontal field angle decreases to about 1/3, although no problem is posed in terms of image quality.
A multi-eye image pickup apparatus for obtaining a high-resolution image by combining the overlapping regions of a plurality of images obtained by phototaking one object using a plurality of image taking systems is disclosed in, e.g., THE JOURNAL OF THE INSTITUTE OF IMAGE ELECTRONICS ENGINEERS OF JAPAN, 90-03-04, pp. 23-28. Also, a multi-eye image pickup apparatus for outputting an image with a large field angle by combining images obtained by coordinates changing processing of a plurality of input images is disclosed in, e.g., Japanese Laid-Open Patent Application No. 5-110926.
With the use of such a multi-eye image pickup apparatus or its application, a multi-eye image pickup apparatus can be obtained, which forms an image having an arbitrary aspect ratio by combining a plurality of images obtained by phototaking one object to be photographed using a plurality of image taking systems with portions of the fields of view of the image taking systems overlapping each other.
As in the multi-eye image pickup apparatus shown in FIG. 7, one object 101 to be photographed is phototaken by using two image taking systems (a left side image taking system 110.sub.L and a right side image taking system 110.sub.R) with portions of the fields of view of the image taking systems overlapping each other, and a left side image P.sub.L obtained by the left side image taking system 110.sub.L and a right side image P.sub.R obtained by the right side image taking system 110.sub.R are combined with each other by a central processing unit (CPU) 120, thereby forming one image P.sub.L+R having an arbitrary aspect ratio.
The above conventional art, however, indicates no method of obtaining an image having a predetermined aspect ratio. Therefore, with the use of the scheme of trimming a combined image, a deterioration in image quality occurs.
In order to solve the above problem, "Multi-eye Image Pickup Apparatus" in Japanese Patent Application No. 5-223544 (U.S. Ser. No. 08/267,117 filed on Jun. 24, 1994) is disclosed which performs a phototaking operation by using a plurality of image taking systems with portions of the fields of view overlapping. In this image pickup apparatus, a plurality of image signals output from the respective image taking systems are combined into one image signal output in a state defined by an arbitrary object distance and the imaging magnification from an image taking system in which the viewpoint position and the optical axis direction are defined by the shift amount of the viewpoint position from each image taking system and the convergence angle of the optical axis. The present applicant also disclosed a method of obtaining an image exhibiting little deterioration in image quality and having undergone correction of distortion caused by convergence.
The above apparatus and method will be described below. FIG. 8 shows the basic arrangement of the above apparatus and method. This apparatus is constituted by two image taking systems, i.e., a right side image taking system 810.sub.R and a left side image taking system 810.sub.L as first and second imaging optical systems.
An object plane 801 is a plane common to the two image taking systems. A left side imaging optical system 811.sub.L and a right side imaging optical system 811.sub.R are imaging optical systems having equivalent specifications. In general, zoom lenses are used as these optical systems. A left side image sensor 812.sub.L and a right side image sensor 812.sub.R also have equivalent specifications. As these sensors, image pick-up tubes such as saticons or solid-state image pick-up elements such as CCDs are used. Optical axes L.sub.L and L.sub.R of these image taking systems are symmetrically inclined by about .theta., respectively, with respect to a normal O-O' to the object plane 801 (a point O is located on the object plane 801) in such a manner that the phototaking fields of view overlap in a predetermined amount in accordance with the selected aspect ratio of a screen. Note that 2.theta. is the convergence angle. Object planes 802.sub.L and 802.sub.R are respectively conjugate to the image sensors 812.sub.L and 812.sub.R and are respectively inclined by .theta. with respect to the object plane 801. Points O.sub.L and O.sub.R are the intersections between the optical axes L.sub.L and L.sub.R and the object plane 801. Points C.sub.L and C.sub.R are the principal points of the imaging optical systems 811.sub.L and 811.sub.R (more specifically, the principal points on the object side). The imaging optical systems 811.sub.L and 811.sub.R include zoom lens groups and focusing lens groups, driving systems for driving these lens groups, and encoders for obtaining position information in the optical axis directions. In addition, each image taking system includes a mechanical system for rotating the image taking system within a plane including the optical axis, a driving system, and an encoder for detecting a rotational angle. A convergence angle controlling system sets a control target value of the convergence angle in accordance with an output signal from each encoder and performs convergence control to obtain an image having a predetermined aspect ratio.
A method of determining a control target value of the convergence angle will be described below with reference to FIGS. 8 and 9.
Let .beta. be the imaging magnification of the left side imaging optical system 811.sub.L and the right side imaging optical system 811.sub.R shown in FIG. 8, z be the object distance (the distance between the point object and the point C.sub.L and the distance between the point object and the point C.sub.R), and 2d be the distance (base line length) between the point C.sub.L and the point C.sub.R. Assume that the viewpoint is set at a point on the normal O-O' separated from the object plane 801 toward the point O' by the distance z, and a virtual plane is set such that the virtual imaging magnification at the viewpoint is .beta.' (i.e., the distance between the viewpoint and the image plane is .beta.'z'). In this case, a temporary imaginary plane I.sub.L+R is obtained by combining an image plane I.sub.L of the left side image sensor 812.sub.L and an image plane I.sub.R of the right side image sensor 812.sub.R, as shown in FIG. 9.
Referring to FIG. 9, points A.sub.L, B.sub.L, C.sub.L, and D.sub.L are the diagonal points of the image plane I.sub.L of the left side image sensor 812.sub.L. These points respectively correspond to points A.sub.L ', B.sub.L ', C.sub.L ', and D.sub.L ' on the temporary imaginary plane I.sub.L+R. Points A.sub.R, B.sub.R, C.sub.R, and D.sub.R are the diagonal points of the image plane I.sub.R of the right side image sensor 812.sub.R. These points respectively correspond to points A.sub.R ', B.sub.R ', C.sub.R ', and D.sub.R ' on the temporary imaginary plane I.sub.L+R. In addition, points E.sub.L and F.sub.L are points on the upper and lower sides of the image plane I.sub.L of the left side image sensor 812.sub.L. These points correspond to the center of the overlapping region. Points E.sub.R and F.sub.R are points on the upper and lower sides of the image plane I.sub.R of the right side image sensor 812.sub.R. These points correspond to the center of the overlapping region. Both the points E.sub.L and E.sub.R correspond to a point E' on the temporary imaginary plane I.sub.L+R. Both the points F.sub.L and F.sub.R correspond to a point F' on the temporary imaginary plane I.sub.L+R.
Assume that the centers of the image planes I.sub.L and I.sub.R are origins, and the horizontal and vertical directions in FIG. 9 are x- and y-axes, respectively. In this case, if coordinate systems are defined on the image planes I.sub.L and I.sub.R, an image point (x.sub.R,y.sub.R) on the image plane I.sub.R of the right side image sensor 812.sub.R corresponds to an image point (x.sub.R ',y.sub.R '), on the temporary imaginary plane I.sub.L+R, which is given by equations (1) and (2): EQU x.sub.R '={(x.sub.R cos .theta.+.beta.z sin .theta.+.beta.d)/(-x.sub.R sin .theta.+.beta.z')}.times..beta.'z' (1) EQU y.sub.R '={y.sub.R /(-x.sub.R sin .theta.+.beta.z')}.times..beta.'z'(2)
In addition, an image point (x.sub.L,y.sub.L) on the image plane I.sub.L of the left side image sensor 812.sub.L corresponds to an image point (x.sub.L ',y.sub.L '), On the temporary imaginary plane I.sub.L+R, which is given by equations (3) and (4): EQU x.sub.L '={(x.sub.L cos .theta.-.beta.z sin .theta.-.beta.d)/(x.sub.L sin .theta.-.beta.z')}.times..beta.'z' (3) EQU y.sub.L '={y.sub.L /(x.sub.L sin .theta.+.beta.z')}.times..beta.'z'(4)
By performing geometrical transform processing like the one indicated by equations (1) to (4), images on a plurality of image sensors, which cause convergence, can be combined into an image on one temporary imaginary plane. Therefore, with the use of an image combining/transform processing portion (not shown) for performing such geometrical transform processing, an image having undergone correction of distortion caused by convergence can be obtained.
In the above method and apparatus, the following problems are posed when an object to be photographed has a distance distribution in which the object includes an object at a greatly different distance from the background from the rest of the objects.
Assume that an object A is located at a distance z.sub.A, as shown in FIG. 8. In this case, if geometrical transform processing based on equations (1) to (4) is performed, the object position on a temporary imaginary plane is shifted by the difference between the object distance z and the object distance z.sub.A. In this case, the positions (x.sub.R ",y.sub.R ") and (x.sub.L ",y.sub.L "), on an ideal temporary imaginary plane for forming a combined image free from any position shift, which correspond to the image point (x.sub.R,y.sub.R) on the image plane of the first image sensor and the image point (x.sub.L,y.sub.L) on the image plane of the second image sensor, respectively, are given by EQU x.sub.R "={[x.sub.R cos .theta.+.beta.z sin .theta.+.beta.d(z/z.sub.A)]/[-x.sub.R sin .theta.+.beta.z(1-z/z.sub.A) cos .theta.+.beta.z'(z/z.sub.A)]}.times..beta.'z' (5) EQU y.sub.R "={y.sub.R /[-x.sub.R sin .theta.+.beta.z(1-z/z.sub.A) cos .theta.+.beta.z'(z/z.sub.A)]}.times..beta.'z' (6) EQU x.sub.L "={[x.sub.L cos .theta.-.beta.z sin .theta.-.beta.d(z/z.sub.A)]/[x.sub.L sin .theta.+.beta.z(1-z/z.sub.A) cos .theta.+.beta.z'(z/z.sub.A)]}.times..beta.'z' (7) EQU y.sub.L "={y.sub.L /[x.sub.L sin .theta.+.beta.z(1-z/z.sub.A) cos .theta.+.beta.z'(z/z.sub.A ]}.times..beta.'z' (8)
Consequently, a positional shift occurs owing to the influence of the object distance. The corresponding shift amounts are respectively given by EQU .DELTA.x.sub.R =x.sub.R "-x.sub.R ', .DELTA.y.sub.R =y.sub.R "-y.sub.R ' .DELTA.x.sub.L =x.sub.L "-x.sub.L ', .DELTA.y.sub.L =y.sub.L "-y.sub.L '(9)
For this reason, an arbitrary point on the object A may be shifted from an ideal position a by (.DELTA.x.sub.R,.DELTA.x.sub.L) in the x direction, and by (.DELTA.y.sub.R,.DELTA.y.sub.L) in the y direction, resulting in a deterioration in the image quality of a combined image.
In addition, since the background is concealed by the object A in the hatched regions in the overlapping region in FIG. 10, these hatched regions are not phototaken by both the first and second image sensors, although the regions are included in the overlapping region. Therefore, a portion IL in FIG. 10 is phototaken as only an image for the first image sensor; and a portion IR, as only an image for the second image sensor. Even if, therefore, images having undergone coordinates changing processing are combined, the object portion may overlap the background portion, resulting in a deterioration in the image quality of the combined image.